Particular embodiments generally relate to a distributed data storage system.
A unit of data, such as a file or object, includes one or more storage units (e.g., bytes), and can be stored and retrieved from a storage medium. For example, disk drives in storage systems are divided into logical blocks that are addressed using logical block addresses (LBAs). The disk drives use spinning disks where a read/write head is used to read/write data to/from the drive. It is desirable to store an entire file in a contiguous range of addresses on the spinning disk. For example, the file may be divided into blocks or extents of a fixed size. Each block of the file may be stored in a contiguous section of the spinning disk. The file is then accessed using an offset and length of the file. The contiguous range of addresses is used because disks are good at sequential access, but suffer performance degradation when random access to different non-contiguous locations is needed.
Storage systems typically do not have a mechanism to minimize the amount of storage used when duplicate copies of data are stored. Duplicate data may occur at different locations within a single file or between different independent files all in the same file system. However, because clients store data based on addresses in the storage medium, duplicate data is typically stored. For example, a first client stores a first file in a first range of addresses and a second client stores a second file in a second range of addresses. Even if duplicate data is found in the first file and the second file, storage systems prefer to store the first file and the second file in separate contiguous locations so that the data for either file can be accessed sequentially.
Some storage systems, such as a write-anywhere file layout (WAFL), a logical volume manager (LVM), or new technology file system (NTFS), allow multiple objects to refer to the same blocks through a tree structure to allow for efficient storage of previous versions. For example, a snapshot feature may eliminate some duplicate data caused by multiple versions of the same file, but this is only to the extent that different versions are created and controlled by the file system itself
Some data storage systems can identify and eliminate duplicate copies of data within or between files. However, these systems typically deal with monolithic systems. For example, the elimination may occur on a single computer system.